Skip to content

fix(evals): a gate policy nested too deep for json is a usage error, not a crash - #1318

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-gate-policy-depth
Oct 24, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-gate-policy-depth

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya/_eval_policy.py::load_policy read the caller's gate policy file with:

    except (OSError, UnicodeError, json.JSONDecodeError) as exc:
        raise evals.EvalError("invalid gate policy %r: %s" % (path, exc)) from exc

RecursionError is now in that tuple, with a docstring naming the exit-code rule it protects. No
signature change, no new error type, no new message format -- a too-deep policy now produces the same
invalid gate policy '<path>': <reason> refusal the other three shapes already produced.

+10/-2 in laya/_eval_policy.py (one word in the tuple, plus the docstring) and +20/-2 in
tests/test_evals.py across three commits (fix(evals):, then two test(evals):).
git diff --numstat 0a05a450 HEAD is exactly those two files.

load_policy has two production call sites -- laya/evals_cli.py run --gate-policy and
compare --gate-policy -- and both are reached before a checkpoint loads, so the reachable surface is
those two flags. Grep of laya/, tests/ and docs/ for load_policy outside this file returns the
definition, those two CLI sites and the tests.

Why

CPython's json does not report a nesting depth it will not walk as a ValueError. It raises
RecursionError -- a RuntimeError, so it passes straight through a handler written for
JSONDecodeError. The caller's file is unreadable, the CLI's own usage-error handler never sees it,
and laya-evals exits 1, the code it keeps for "a threshold or a baseline tolerance failed".

Measured by running the CLI in a child process, printing which laya/evals_cli.py it imported before
answering (laya-bench/probe_evals_policy_depth.py; base witness
laya-bench/probe_evals_policy_depth.BASE.log 14:23Z against a pristine 0a05a450 worktree, branch
witness laya-bench/probe_evals_policy_depth.BRANCH.log 14:23Z against this tree). The policy file is
"[" * 100000 + "]" * 100000:

arm 0a05a450 this branch
run --gate-policy <too deep> TRACEBACK, exit 1 — RecursionError: Stack overflow (used 8144 kB) while decoding a JSON array from a unicode string usage error, exit 2 — laya-evals: invalid gate policy '<path>': Stack overflow (used 8144 kB) …
compare --gate-policy <too deep> TRACEBACK, exit 1 usage error, exit 2
run --gate-policy {not json} exit 2 exit 2, byte-identical message
run --gate-policy <absent path> exit 2 exit 2, byte-identical message
run --gate-policy <valid, unsatisfiable> exit 1, FAIL: slice gate rule 1 (language=en, noul_accuracy): metric or slice is missing identical
load_policy("<too deep>") in-process raises RecursionError raises EvalError

The two changed rows are the bug; the four others are the controls that show nothing else moved. The
valid-policy row is the one a reviewer should notice: a genuine quality failure still exits 1 with the
same FAIL: line, so this does not convert failures into usage errors.

Why a CI job cannot tell the two apart: exit 1 is the code "the model regressed" already prints, and a
traceback on stderr from a job that reads only the code looks like a scored run that failed its gate. A
policy file truncated mid-write by a crashed generator, or an editor that dumped a deeply-merged
rules array, produces that. docs/evals.md:497-498 states the rule this restores, verbatim:

Invalid policy syntax exits 2 before a checkpoint loads; a quality failure exits 1.

and docs/evals.md:28-29 gives the same contract for the whole CLI. The _load_report reader in
laya/evals_cli.py:294 and the agreement-run readers at laya/evals_cli.py:590 already catch
RecursionError for exactly this reason; this makes the third caller-supplied JSON door in the harness
agree with the two that were fixed.

The tests add two things to test_cli_rejects_invalid_policy_json_and_version, which already walked
{not json and a version: 2 document and asserted only the exit code:

  1. a third arm for unclosable depth, asserting no Traceback on stderr, that the refusal still starts
    with laya-evals: , and that it says gate policy -- so the message says which flag broke;
  2. an absent path, asserting the same. OSError was already in the tuple, and main has its own
    FileNotFoundError arm, so this arm passes on main too; it is here because the mutation harness
    found that nothing proved the loader rather than main was answering, and the two messages differ
    in the one part that matters (which of the CLI's several paths was missing).

How it was verified

VENV="/Users/aashish/github contribution /GIT/laya/.venv/bin/python"
TREE="/Users/aashish/github contribution /GIT/laya-policy-depth"
cd "$TREE"   # the venv's editable install points at a different checkout, so every run pins the tree

The lane CI runs. tests/test_evals.py is not in scripts/test_suites.py; CI runs it as its own
step, .github/workflows/ci.yml:174 =
python -m pytest tests/test_evals.py tests/test_evals_agreement.py -q. That exact command, offline,
on this tree: 264 passed (laya-bench/ci_step_evals_policy_branch_head7b7f19d6.log, at this PR's
final head; laya-bench/ci_step_evals_policy_branch_head702595cb.log 14:23Z is the same count at the
fix commit) against 264 passed on a pristine 0a05a450 worktree
(laya-bench/ci_step_evals_ctrl0a05.log, 14:18Z). The count does not move, and that is expected:
both new arms live inside an existing test function, so collection cannot show them. The revert arm
below is the proof they execute.

Revert arm. branch tests + git show 0a05a450:laya/_eval_policy.py ->
1 failed, 263 passed (laya-bench/revert_arm_evals_policy_head7b7f19d6.log, 14:41Z), and the failure
is named: tests/test_evals.py::test_cli_rejects_invalid_policy_json_and_version, with
E RecursionError: Stack overflow (used 8144 kB) while decoding a JSON array from a unicode string as
the line that fails it. Base file sha256 9788b90c673f, branch 729d64e76b90; restored byte-for-byte
(post-restore sha256 729d64e76b90, git status --porcelain empty). The other 263 -- every
pre-existing gate-policy arm, both run and compare -- pass on base, so nothing a caller could rely
on changed.

Mutation harness (laya-bench/mutate_evals_policy_depth.py; first pass
laya-bench/mutate_evals_policy_depth.log 14:26Z, second pass
laya-bench/mutate_evals_policy_depth.pass2.log 14:38Z). Predictions written into the harness before
the first run; baseline 185 passed, each mutant a full literal block compile()-checked, file
restored and sha256-verified between mutants.

mutant claim pass 1 pass 2
m1 RecursionError dropped the bug reappears KILLED (1) KILLED (1)
m2 RecursionError replaced by RuntimeError — SURVIVED SURVIVED
m3 json.JSONDecodeError dropped {not json escapes KILLED (1) KILLED (1)
m4 OSError dropped absent file unhandled SURVIVED KILLED (1)
m5 message loses the "invalid gate policy" kind refusals become unattributable KILLED (1) KILLED (1)
m6 raises ValueError, not EvalError main's handler misses it KILLED (1) KILLED (1)
m7 docstring demoted to a local variable text-only edit SURVIVED SURVIVED

Pass 2 result: 7 mutants, 5 killed, 2 survived, 1 predicted-count mismatch. All three first-pass
survivors are disclosed rather than dropped:

  • m4 became killable and I killed it. Dropping OSError still exited 2, because main
    (laya/evals_cli.py) has its own narrowed FileNotFoundError arm -- so the surviving mutant was an
    equivalence at exit-code level, not a gap in my change. It is not equivalent at message level:
    main prints only the OS error, so the caller loses which flag pointed at the missing file. Test arm
    2 above pins that, and m4 then fails on exactly one test (predicted 0 failing, got 1 -- the
    mismatch is the pin working, disclosed as such).
  • m2 is equivalent by inheritance, not a coverage gap. issubclass(RecursionError, RuntimeError)
    is True in CPython 3.14, so RuntimeError in the tuple catches every input RecursionError
    catches and the refusal is byte-identical. No distinguishing input exists. I kept
    RecursionError because it says which failure is being handled; the mutant survives because there is
    nothing for a test to see.
  • m7 is the deliberate positive control: a non-behavioural edit must not be counted by this
    ledger, and it isn't.

Gate. LAYA_TREE=$TREE HF_HUB_OFFLINE=1 $VENV -B laya-bench/ci_gate.py at this branch's head ->
laya-bench/gate_evals_policy_head7b7f19d6.log, diffed against the pristine 0a05a450 control
(laya-bench/gate_ctrl0a05.log, 14:10Z) with laya-bench/diff_gate_logs.py. The shared 92-suite list
structurally cannot see this change (tests/test_evals.py is not in it), which is why the CI step and
the revert arm above are the primary witnesses; the gate is here to show the change breaks nothing that
CI does run. The gate at the preceding head 792b548e read 81 invocations, all 92 suites passed,
ci_gate exit=0
(laya-bench/gate_evals_policy_head792b548e.log, 14:29Z, with tree= and
laya_from= both naming this tree), and its diff against the control is
invocations: branch=81 control=81, identical: 80 differing: 1
(laya-bench/gate_evals_policy_diff_head792b548e.txt, 14:29Z). The one differing invocation is
tests/test_mcp.py, where the same set members print in a different order
({'routing', 'latency_ms', 'answers'} vs {'answers', 'latency_ms', 'routing'}) — pre-existing
nondeterminism in a suite this PR does not touch, and both tails read all mcp tests passed. Nothing
gained or lost a failure.

Docs. No page moves. docs/evals.md:497-498 already states the rule, and the docstring added here
cites it in prose rather than publishing new contract text. docs/reference/*.md has no griffe
directive naming laya._eval_policy (grep of the ::: laya directives returns laya.confidence
symbols), so the new docstring paragraph is not published. The strict docs build is not run:
/tmp/docsvenv (the zensical venv used for prior doc sweeps) no longer exists, and git diff --numstat
covers only laya/_eval_policy.py and tests/test_evals.py, which is the narrower proof.

Lint, compile, secrets. /opt/homebrew/bin/ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 -> All checks passed!. $VENV -m compileall -q laya/ tests/ -> rc=0. The +
lines of git diff 0a05a450 HEAD (30 lines) scanned for
api_key|secret|token|password|passwd|credential|authorization|bearer |private key|ghp_|gho_|sk-…|xox[baprs]-|AKIA…:
0 matches, rc=1.

Duplicate guard. 30 open PRs at 14:43Z (gh pr list --repo NandhaKishorM/laya --state open --limit 200 --json files), main still 0a05a450. Filtering every open PR's file list for
laya/_eval_policy.py, laya/evals.py, laya/evals_cli.py, laya/evidence.py and
tests/test_evals.py returns one hit -- my own #1317, which is a different unit -- and no other
editor of laya/_eval_policy.py.

Cross-PR conflict, measured not assumed. #1317, this PR and the dataset-reader PR all insert into
tests/test_evals.py, so git merge-tree --write-tree was run for all six orderings of the three
branch pairs: every one exits 0 with no conflicted path. laya/_eval_policy.py has one editor in
the stack: this PR.

Cross-port check. Grep of laya-java, laya-go, laya-dotnet, laya-ts, laya-rust for a
gate-policy JSON reader returns 0 files, so no port fixture pins the current escape.

Notes for the reviewer

The depth limit itself is untouched. This does not raise or lower how deep json will go, and does not
add a max_depth knob; CPython's scanner still answers at ~8 MB of stack, and its message is what the
refusal now carries. laya/evals.py::Dataset.from_jsonl has the same shape of hole for the dataset
file and is deliberately not in this PR (it needs its own reader-level tests, and #1317/#this already
share tests/test_evals.py).

main's FileNotFoundError arm stays as it is: it is the second door for a path that was never
opened, and the two arms differ in that load_policy names which flag asked for the file.

No new dependency, no hosted service, no public API signature change, no change for any policy that
json can read.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (0a05a450; git rev-parse refs/remotes/upstream/main = 0a05a450)
  • Tests pass locally (the CI eval-harness step 264 passed, same as base, plus a revert arm that names the new test and a 7-mutant ledger with all three survivors explained)
  • Docs or examples updated when the public API changed (the public API did not change; docs/evals.md:497-498 already documents the exit-code rule this restores, so nothing needed editing)

Fixes #

…the flag

The mutation harness for this change found that dropping `OSError` from the `except` tuple still
exited 2 -- `main` has its own `FileNotFoundError` arm -- so nothing in the suite proved the loader
catches it. It is worth pinning because the two messages differ where it matters: the loader says
which flag pointed at the missing file, and `main`'s fallback names only the path.
@NandhaKishorM
NandhaKishorM merged commit bc897d6 into NandhaKishorM:main Oct 24, 2026
48 of 52 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants